Processing captured images having geolocations

ABSTRACT

Methods ( 105 ), apparatuses ( 600 ), and computer readable storage mediums for processing captured images having geolocations related to the captured images at the time of capture are disclosed. A representative geolocation is associated ( 110 ) with each group of images previously captured at the same location. For each representative geolocation, based on at least the timestamp of the images associated with the representative geolocation, the probability is determined ( 120 ) that a further image will be captured at or near the representative geolocation. For representative geolocations with a determined probability above a predetermined threshold, the respective representative geolocation is associated ( 130 ) with at least one personal place.

REFERENCE TO RELATED PATENT APPLICATION

This application claims the benefit under 35 U.S.C. §119 of the earlierfiling date of Australian Patent Application No 2009243486, filed 2 Dec.2009, hereby incorporated by reference in its entirety as if fully setforth herein.

TECHNICAL FIELD

The present invention relates to image selection and in particular tothe use of metadata associated with photographs.

BACKGROUND

Using the Internet as a medium for sharing photos is becoming widelypopular due to the immediacy and extensive reach to audiences. Themethods of sharing such photos are increasingly more sophisticated,ranging from sending email messages with photos attached to specificrecipients to uploading photos to photo sharing and social networkingsites. Consequently, photos can be shared to varying extents, rangingfrom a single trusted recipient to virtually everyone with access to theInternet, including strangers who may have other agendas besidesappreciating the photos.

Depending on the content of the photos, public sharing of photos may bea privacy concern to a user, because the photos may show, for example,the inside of a private residence and possibly children playing.Technological advances that enable capturing of information related aphoto and storing such information in the corresponding metadataheightens such concerns. For example, the geolocation of where a photois captured can be stored in the metadata by a device equipped with abuilt-in GPS receiver, allowing the subject matter of the photo to beassociated with the location at which the subject matter was captured.In addition, a device with a face-recognition function enables automatictagging of faces appearing in a photo with corresponding names andstores the tags in the metadata.

Photo sharing service providers are attempting to address theabove-mentioned concerns by offering various security measures as partof the sharing process. Depending on the photo sharing site, thesecurity measures may range from providing a global setting for enablingor disabling the sharing of geotags (geolocation tags) for the entirephoto collection of a user, to providing a per photo setting for sharinga geotag with selected groups of people, e.g. friends, family, everyone,etc. Even though users have a mechanism to control the sharing ofgeotags of their photos, the actual task can be laborious and tediousdue to the ever-increasing size of their photo collections. To share thegeotags of a user's photo collection fully, the user is required toinspect the user's photo collection and make an appropriate setting foreach photo accordingly. The time-consuming and potentially error-pronenature of this process would deter users from sharing the geotags oftheir photo collection as much as possible, leaving the geotags of manyphotos unshared even without the risk of any privacy concern. Often, theprocess causes inadvertent sharing of geotags of private places for manyusers. The practice of photo sharing service providers of making publicsharing of geotags a default option exacerbates this situation. Thiscauses users to opt in the scheme without necessarily considering theimplications.

Users are in need of a more effective method that enables the users tocontrol the sharing of geotags and other metadata of their photosindividually, without requiring the users to inspect the photos to makea determination.

SUMMARY

In accordance with an aspect of the invention, there is provided amethod of processing captured images having geolocations related to thecaptured images at the time of capture. A representative geolocation isassociated with each group of images previously captured at the samelocation. For each representative geolocation, based on at least thetimestamp of the images associated with the representative geolocation,the probability is determined that a further image will be captured ator near the representative geolocation. For representative geolocationswith a determined probability above a predetermined threshold, therespective representative geolocation is associated with at least onepersonal place.

The method may further comprise the step of capturing an image having ageolocation related to the captured image at the time of capture.

The method may further comprise the step of suppressing at least onemetadata value of a captured image if the geolocation related to thecaptured image is at or near at least one personal place. The at leastone metadata value may be suppressed by at least one of storing themetadata value in an external data store separately from the capturedimage and encrypting the metadata value.

The method may further comprise the step of specifying a geolocation asa personal location. Specifying the geolocation of a personal locationmay comprise inputting textual addresses, or importing addresses from acontact list, or manually entering the geolocation.

The captured image may be captured using an image capturing apparatusadapted to acquire a geolocation related to the captured image at thetime of image capture.

The geolocations and at least the timestamps of the images may be storedfor further processing, the images being downloaded separately.

In accordance with another aspect of the invention, there is provided anapparatus for processing captured images having geolocations related tothe captured images at the time of capture. The apparatus comprises: amemory for storing data and a computer program; and a processor unitcoupled to the memory for executing a computer program, the memory andthe processor configured to process the captured images. The computerprogram comprises: a computer program code module for associating arepresentative geolocation with each group of images previously capturedat the same location; a computer program code for each representativegeolocation, determining, based on at least the timestamp of the imagesassociated with the representative geolocation, the probability that afurther image will be captured at or near the representativegeolocation; and a computer program code module for representativegeolocations with a determined probability above a predeterminedthreshold, associating the respective representative geolocation with atleast one personal place.

The apparatus may further comprise a computer program code module forsuppressing at least one metadata value of a captured image if thegeolocation related to the captured image is at or near at least onepersonal place. The at least one metadata value may suppressed by atleast one of storing the metadata value in an external data storeseparately from the captured image and encrypting the metadata value.

The apparatus may further comprise an image capturing apparatus adaptedto acquire a geolocation related to a captured image at the time ofimage capture.

The geolocations and at least the timestamps of the images may be storedin the memory of the apparatus for further processing, allowing theimages to be downloaded from the apparatus and removed from the memory.

In accordance with yet another aspect of the invention, there isprovided a computer readable storage medium having recorded therein acomputer program for processing captured images having geolocationsrelated to the captured images at the time of capture for execution by aprocessing unit. The computer program comprises: a computer program codemodule for associating a representative geolocation with each group ofimages previously captured at the same location; a computer program codemodule for each representative geolocation, determining, based on atleast the timestamp of the images associated with the representativegeolocation, the probability that a further image will be captured at ornear the representative geolocation; and a computer program code modulefor representative geolocations with a determined probability above apredetermined threshold, associating the respective representativegeolocation with at least one personal place.

The computer readable storage medium may further comprise a computerprogram code module for suppressing at least one metadata value of acaptured image if the geolocation related to the captured image is at ornear at least one personal place.

The at least one metadata value may be suppressed by at least one ofstoring the metadata value in an external data store separately from thecaptured image and encrypting the metadata value.

The images may be captured using an image capturing apparatus adapted toacquire a geolocation related to a captured image at the time of imagecapture.

The geolocations and at least the timestamps of the images may be storedfor further processing, allowing the images to be downloaded.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the invention are described hereinafter with reference tothe following drawings, in which:

FIG. 1 is a schematic flow diagram illustrating a method of processingcaptured images having geolocations related to the captured images atthe time of capture according to an embodiment of the invention, theprocessed images can be used to selectively suppressing at least onemetadata value of an image captured by an image capturing apparatus;

FIG. 2 is a schematic flow diagram illustrating a method of calculatingthe probability of when a further image will be captured at a particularlocation, as used in the method of FIG. 1;

FIG. 3 is a schematic flow diagram illustrating a method of suppressingmetadata as used in the method of FIG. 1 according to a furtherembodiment of the invention;

FIG. 4 is a schematic flow diagram illustrating a method of suppressingmetadata as used in the method of FIG. 1 according to a still furtherembodiment of the invention;

FIG. 5A and FIG. 5B are plots illustrating examples of predictionintervals as calculated using the method of FIG. 2;

FIG. 6A is a cross-section diagram of a camera system on which theembodiments of the invention may be practised;

FIG. 6B is a schematic block diagram of an image capture system on whichthe embodiments of the invention may be practised; and

FIG. 7 is a flow diagram illustrating a method of processing of capturedimages having geolocations in accordance with an alternative embodimentof the invention.

DETAILED DESCRIPTION

Methods, apparatuses, and computer program products for processingcaptured images having geolocations related to the captured images atthe time of capture are disclosed. In the following description,numerous specific details, including particular types of metadata,prediction intervals, and the like are set forth. However, from thisdisclosure, it will be apparent to those skilled in the art thatmodifications and/or substitutions may be made without departing fromthe scope and spirit of the invention. In other circumstances, specificdetails may be omitted so as not to obscure the invention.

Where reference is made in any one or more of the accompanying drawingsto steps and/or features, which have the same reference numerals, thosesteps and/or features have for the purposes of this description the samefunction(s) or operation(s), unless the contrary intention appears.

[Introduction]

The embodiments of the invention are aimed at enabling users to sharetheir photo collections, including associated metadata, as fully aspossible without the unacceptable risk of revealing privacy or sensitivepersonal information. Obvious privacy concerns include the publicdisclosure of geotags (the geolocation of where a photo was taken by adevice equipped with, for example, a built-in GPS receiver) of photostaken at a private residence or school, etc. The captured image may becaptured using an image capturing apparatus adapted to acquire ageolocation related to the captured image at the time of image capture.Other less obvious but still important privacy concerns include thepublic disclosure of geotags of places that are general public, but maybe considered as personal by a user due to frequent and predictablevisits, e.g. local parks, sports grounds, and the like. Thus, ingeneral, a user may consider places such as local parks, entertainmentvenues, schools, sports grounds, and the like as “personal” locations. Auser's home may be even more personal and be deemed a “private”location.

The general approach of the embodiments of the invention involvesdetermining and selecting a subset of images (e.g., photos) from a photocollection that may constitute a privacy concern for a user if certainassociated metadata is shared with a corresponding image. In thefollowing description, for ease of description only, the term “photo”will be used rather than the term “image”, however, it will apparent tothose skilled in the art in the light of this disclosure that theinvention is not limited simply to photos. Several embodiments aredescribed hereinafter for selecting the subset of photos and suppressingthe metadata of the photos, so that the risk of revealing privacyinformation is mitigated. The embodiments of the invention are able toprocess and suppress metadata of sensitive images on a per image (orphoto) basis based on geolocation. This is in marked contrast tosuppressing metadata of an image dependent upon the recipient. Further,this method is able to selectively suppress what metadata is suppressed.

[Image Capture System]

Preferably, the method is implemented in an image capturing apparatus asshown in FIGS. 6A and 6B. FIG. 6A is a cross-section diagram of anexemplary image capturing apparatus 600, upon which the variousarrangements described can be practiced. In the general case, the imagecapture system 600 is a digital still camera or a digital video camera(also referred to as a camcorder).

As seen in FIG. 6A, the camera system 600 comprises an optical system602, which receives light from a scene 601 and forms an image on asensor 621. The sensor 621 comprises a 2D array of pixel sensors thatmeasure the intensity of the image formed on the array of pixel sensorsby the optical system 602 as a function of position. The operation ofthe camera 600, including user interaction and all aspects of reading,processing and storing image data from the sensor 621 is coordinated bya main controller 622, which comprises a special-purpose computersystem. This system is considered in detail hereinafter with referenceto FIG. 6B.

The user can communicate with the controller 622 via a set of buttonsincluding a shutter release button 628, used to initiate focus andcapture of image data, and other general and special purpose buttons624, 625, 626, which may provide direct control over specific camerafunctions such as a flash operation or support interaction with agraphical user interface presented on a display device 623 (e.g., an LCDpanel). The display device 623 may also have a touch screen capabilityto further facilitate user interaction. Using the buttons 624, 625, 626and controls, the behaviour of the camera 600 can be controlled ormodified. Typically, capture settings such as the priority of shutterspeed or aperture size when achieving a required exposure level, thearea used for light metering, use of flash, ISO speed, options forautomatic focusing and many other photographic control functions can becontrolled. Further, processing options such as the colour balance orcompression quality can be controlled. The display device 623 istypically also used to review the captured image or video data.Commonly, a still image camera uses the display device 623 to provide alive preview of the scene, thereby providing an alternative to anoptical viewfinder 627 for composing prior to still image capture andduring video capture.

The optical system 602 comprises an arrangement of lens groups 610, 612,613 and 617, which can be moved relative to each other along a line 631parallel to an optical axis 603 under control of a lens controller 618.Such movement of the lens group elements 610, 612, 613 and 617 canachieve a range of magnification levels and focus distances for theimage formed at the sensor 621. The lens controller 618 may also controla mechanism 611 to vary the position, on any line 632 in the planeperpendicular to the optical axis 603, of a corrective lens group 612,in response to input from one or more motion sensors 615, 616 or thecontroller 622, so as to shift the position of the image formed by theoptical system 602 on the sensor 621. Typically, the corrective opticalelement 612 is used to effect an optical image stabilisation bycorrecting the image position on the sensor 621 for small movements ofthe camera 600, such as those caused by hand-shake. The optical system602 may further comprise an adjustable aperture 614 and a shuttermechanism 620 for restricting the passage of light through the opticalsystem 602. Although both the aperture 614 and shutter 620 are typicallyimplemented as mechanical devices, the aperture 614 and shutter 620 mayalso be constructed using materials, such as liquid crystal, whoseoptical properties can be modified under the control of an electricalcontrol signal. Such electro-optical devices have the advantage ofallowing both shape and the opacity of the aperture 614 to be variedcontinuously under control of the controller 622.

FIG. 6B is a schematic block diagram of the main controller 622 of FIG.6A, in which other components of the camera 600 that communicate withthe controller 622 are depicted as functional blocks. In particular, animage sensor 691 and a lens controller 698 are depicted withoutreference to their physical organisation or the image forming process.The image sensor 691 and the lens controller 698 are treated as devicesthat perform specific pre-defined tasks and to which data and controlsignals can be passed. FIG. 6B also depicts a flash controller 699,which is responsible for operation of a strobe light that can be usedduring image capture in low light conditions as auxiliary sensors 697,which may form part of the camera system 600. Auxiliary sensors 697 mayinclude:

-   -   orientation sensors that detect if the camera is in a landscape        or a portrait orientation during image capture;    -   motion sensors that detect movement of the camera;    -   a GPS receiver that obtains a current geolocation to provide a        geotag;    -   other sensors that detect the colour of the ambient illumination        or assist with autofocus and so on.        Although these are depicted as part of the controller 622, the        auxiliary sensors 697 may be implemented as separate components        within the camera system 600.

The controller 622 comprises a processing unit 650 for executing programcode, Read Only Memory (ROM) 660 and Random Access Memory (RAM) 670, aswell as non-volatile mass data storage 692. In addition, at least onecommunications interface 693 is provided for communication with otherelectronic devices and network services such as printers, displays,general purpose computers and the Internet. Examples of communicationinterfaces include USB, IEEE1394, HDMI, Ethernet and Wi-Fi. An audiointerface 694 comprises one or more microphones and speakers for captureand playback of digital audio data. A display controller 695 and abutton interface 696 are also provided to interface the controller 622to the physical display 623 and controls 624, 625, 626 present on thecamera body. The foregoing components of the controller 622, the imagesensor 691, and the lens controller 698 are interconnected by a data bus681 and a control bus 682.

In a capture mode, the controller 622 operates to read data from theimage sensor 691 and the audio interface 694 and manipulates that datato form a digital representation of the scene that can be stored to thenon-volatile mass data storage 692. In the case of a still image camera,image data may be stored using a standard image file format such as JPEGor TIFF, or the image data may be encoded using a proprietary raw dataformat that is designed for use with a complimentary software productthat provides conversion of the raw format data into a standard imagefile format. Such software is typically run on a general-purposecomputer. For a video camera, the sequences of images that comprisecaptured video are stored using a standard format such DV, MPEG, H.264.Some of these formats such as AVI or Quicktime are organised into filesreferred to as container files. Other formats such as DV, which arecommonly used with tape storage, are written as a data stream. Thenon-volatile mass data storage 692 is used to store the image or videodata captured by the camera system 600 and has a large number ofrealisations including but not limited to removable flash memory such asa compact flash (CF) or secure digital (SD) card, memory stick,multimedia card, miniSD or microSD card, optical storage media such aswritable CD, DVD or Blu-ray disk, or magnetic media such as magnetictape or hard disk drive (HDD) including very small form-factor HDDs suchas microdrives. The choice of mass storage depends on the capacity,speed, usability, power and physical size requirements of the particularcamera system.

In a playback or preview mode, the controller 622 operates to read datafrom the mass storage 692 and presents that data using the display 695and audio interface 694.

The processor 650 can execute programs stored in one or both of theconnected memories 660 and 670. When the camera system 600 is initiallypowered up system program code 661, resident in ROM memory 660, isexecuted. This system program permanently stored in the camera system'sROM 660 is sometimes referred to as firmware. Execution of the firmwareby the processor 650 fulfils various high-level functions, includingprocessor management, memory management, device management, storagemanagement and user interface.

The processor 650 includes a number of functional modules including acontrol unit (CU) 651, an arithmetic logic unit (ALU) 652, a digitalsignal processing engine (DSP) 653 and a local or internal memorycomprising a set of registers 654 which typically contain atomic dataelements 656, 657, along with internal buffer or cache memory 655. Oneor more internal buses 659 interconnect these functional modules. Theprocessor 650 typically also has one or more interfaces 658 forcommunicating with external devices via the system data 681 and control682 buses using a connection 655.

The system program 661 includes a sequence of instructions 662 though663 that may include conditional branch and loop instructions. Theprogram 661 may also include data which is used in execution of theprogram. This data may be stored as part of the instruction or in aseparate location 664 within the ROM 660 or in RAM 670.

In general, the processor 650 is given a set of instructions, which areexecuted therein. This set of instructions may be organised into blocksthat perform specific tasks or handle specific events that occur in thecamera system. Typically the system program 661 waits for events andsubsequently executes the block of code associated with that event. Thismay involve setting into operation separate threads of execution runningon independent processors in the camera system such as the lenscontroller 698 that subsequently executes in parallel with the programrunning on the processor 650. Events may be triggered in response toinput from a user as detected by the button interface 696. Events mayalso be triggered in response to other sensors and interfaces in thecamera system.

The execution of a set of the instructions may require numeric variablesto be read and modified. Such numeric variables are stored in RAM 670.The disclosed method uses input variables 671, that are stored in knownlocations 672, 673 in the memory 670. The input variables are processedto produce output variables 677, that are stored in known locations 678,679 in the memory 670. Intermediate variables 674 may be stored inadditional memory locations in locations 675, 676 of the memory 670.Alternatively, some intermediate variables may only exist in theregisters 654 of the processor 650.

The execution of a sequence of instructions is achieved in the processor650 by repeated application of a fetch-execute cycle. The control unit651 of the processor 650 maintains a register called the program counterwhich contains the address in memory 660 of the next instruction to beexecuted. At the start of the fetch execute cycle, the contents of thememory address indexed by the program counter is loaded into the controlunit. The instruction thus loaded controls the subsequent operation ofthe processor 650, causing for example, data to be loaded from memoryinto processor registers 654, the contents of a register to bearithmetically combined with the contents of another register, thecontents of a register to be written to the location stored in anotherregister and so on. At the end of the fetch execute cycle the programcounter is updated to point to the next instruction in the program.Depending on the instruction just executed this may involve incrementingthe address contained in the program counter or loading the programcounter with a new address in order to achieve a branch operation.

Each step or sub-process in the processes of flow charts are associatedwith one or more segments of the program 661 and is performed byrepeated execution of a fetch-execute cycle in the processor 650 orsimilar programmatic operation of other independent processor blocks inthe camera system.

The process of determining whether a location is regarded as personal bya user when capturing a photo is now described in detail hereinafter.This process involves analysing previously captured photos to identify apattern exhibited by the user when capturing photos at each location.The frequency and regularity of the pattern is used to assess thepersonal nature of a location.

[Processing Images Having Related Geolocations]

The method 100 of FIG. 1 processes captured images having geolocationsrelated to the captured images at the time of capture according to anembodiment of the invention. This processing of captured images isdenoted by a dashed box 105 comprising three steps 110, 120, 130. Theresult of the processed images can be used to selectively suppress atleast one metadata value of an image captured by an image capturingapparatus using steps 140, 150, and 160 of the method 100. Images (e.g.,photos) captured at locations can be identified that may be consideredas personal to a user and selected metadata values of the identifiedimages, which the user may be sensitive about disseminating, can besuppressed. For example, metadata such as geotags and name tags ofpeople may be suppressed. However, other types of metadata may besuppressed. The method 100 starts at step 110.

In step 110, a representative geolocation is associated with each groupof images captured at the same location. This step involves analysingmetadata of previously captured images, including their geotags. In oneembodiment, the previously captured images and metadata are storedtogether. Groups of images are identified such that the images within agroup have all been captured within close proximity of each other at aparticular location, e.g. at home, a park, etc. For example, anextension of the Quality Threshold (QT) clustering algorithm may be usedto determine the groups, by:

(i) choosing a maximum diameter for the clusters to be formed, typically100 metres;

(ii) building a candidate cluster for each image such that with thegeolocation of the image as the centre of the cluster, all images in thecandidate cluster are within the specified maximum diameter;

(iii) saving the candidate cluster with the most number of images as afinal cluster and removing the images for further consideration;

(iv) recursively finding the next cluster with the most number of imagesfrom the remaining images until all of the images are in a finalcluster.

As an extension, if two or more final clusters are separated by adistance less than the specified maximum diameter, they are combinedinto a single cluster. Each of the final clusters represents a group ofimages captured within close proximity of each other at a particularlocation.

For each of the groups of images identified, a representativegeolocation derived from the geotags of the images within the group isassociated with the group. The representative geolocation may beselected from the geotags of the images in the group such that therepresentative geolocation is the overall nearest neighbour to theremaining images, that is, the sum of the distances from the remainingimages of the group is the shortest. In other circumstances, therepresentative geolocation may be calculated from the geolocations of atleast some of the images within the group.

At step 120, for each group, the probability that a further image willbe captured at or near the corresponding representative geolocation isdetermined. This step is described hereinafter with reference to FIG. 2.

In step 130, if the probability determined in step 120 is above apredetermined threshold, the corresponding location is classified aspersonal. The respective representative geolocation is associated with apersonal place. The processing of captured images having geolocationsrelated to the captured images at the time of capture is completed atthe end of step 130. The remaining steps of FIG. 1 can be practiced inaccordance with a further embodiment of the invention to suppressselected metadata of a captured image dependent upon the processing 105carried out in steps 110, 120, 130.

In step 140, an image is captured using an image capturing apparatus.The main controller 622 can control capture of an image as a result ofthe user operating the camera 600 to capture the image.

In decision step 150, a check is made to determine if the image has beencaptured at a personal location. This involves determining if thegeolocation of the captured image is considered as personal by lookingup a representative geolocation classified as personal in step 130 thatis within close proximity to the current geolocation. If step 150determines that the current geolocation is not personal (NO), theprocess 100 ends at step 199 without suppression of metadata. However,if step 150 determines that one is found (YES) indicating the currentgeolocation is considered as personal, processing continues at step 160.In step 160, selected metadata that a user may consider to be sensitive(i.e. personal or private) is suppressed in respect of the capturedimage. The immediate suppression of selected metadata, such as geotag,date and face tags of recognised faces, straight after capturingprovides the benefit that images may be safely disseminated withoutdisseminating sensitive metadata of a personal or private nature. Anumber of metadata suppression embodiments that can be practice in step160 are described hereinafter. Processing then ends at step 199.

The foregoing embodiments of the invention are described as a singlelinear process 100 with minimal branching for simplicity and a completeunderstanding. After gaining an understanding of the method 100, itshould become apparent that the personal location analysis stage 105,including steps 110 to 130, and the image capturing stage, comprisingsteps 140 to 160, are two distinct stages and may be performedseparately with minimal dependence. For efficiency, the personallocation analysis stage 105 may be processed in advance of the imagecapturing stage to avoid any delays in starting the latter. Havingprocessed the personal location analysis stage 105 at least once, theimage capturing stage may be processed repeatedly to capture multipleimages without further processing the personal location analysis stage.In addition, newly captured images may be added to the original imagecollection for further personal location analysis on an incrementalbasis using the saved results from the previous analysis. The benefit ofincluding the latest images in the personal location analysis is thatthe classification of personal locations is able to adapt to the currentimage-capturing pattern of the user and modify the classification asnecessary. That is, if photos are being taken more regularly at aparticular location as a result of more regular visits by a user, theclassification of the location may change from non-personal to personalor vice versa.

Once the captured images have been analysed in the personal locationanalysis stage 105, the images are no longer required when performingfurther analysis on newly captured images if the data of the capturedimages used for the analysis, e.g. timestamps, and the results of theanalysis are saved, e.g. in persistent memory 664. The data can beretrieved for further personal location analysis without the originalimages. This allows the captured images to be downloaded from the camerasystem 600 to relinquish space on the mass storage 692 for storingfurther images.

The main controller 622 can implement step 110 and the previouslycaptured images and metadata can be stored in the camera 600, e.g. inthe mass storage 692. In step 130, the main controller 622 may performthe classifying of whether a representative geolocation is considered aspersonal to the user according to the thresholds for the user, and thethresholds may be stored in persistent memory 664. In step 150, thecontroller 622 can determine if the location of the captured imageobtained by an auxiliary sensor 697, such as a GPS receiver, isconsidered as personal by looking up a representative geolocationclassified as personal in step 130 that is within close proximity to thecurrent geolocation. Step 160 may be performed in the camera system 600.The immediate suppression of selected metadata, such as geotag, date andface tags of recognised faces, straight after capturing provides thebenefit that images may be safely uploaded to the Internet for instantsharing via the communication interface 693. Steps 110, 120, 130 may beperformed in the camera 600, or in a general-purpose computer separatefrom the camera. For example, steps 110, 120, 130 may be performed on ageneral-purpose camera, but the remaining steps 140, 150, 160 may beperformed on the camera using the results of steps 110, 120, 130.However, steps 110-130, 150, and 160 may be performed on the camera orthe general-purpose computer.

[Determining Probability about Further Image]

FIG. 2 shows in detail the step 120 of FIG. 1 performed for each groupof images. At step 210, a sequence of chronological timestamps {t₁, t₂,. . . , t_(m)} grouped by date is obtained from the images in a group,i.e. from the metadata of the images. In the sequence of chronologicaltimestamps {t₁, t₂, . . . , t_(m)}, t_(i) is the date when an image wascaptured, e.g. {18/7/2009, 25/7/2009, 2/8/2009, 8/8/2009, 15/8/2009,29/9/2009}. In step 220, the time duration in days {x₁, x₂, . . . ,x_(n)} is calculated between the obtained successive timestamps, wheren=m−1 and x_(i) is the time duration between t_(i) and t_(i+1), e.g. {7,8, 6, 7, 14}. In step 230, the prediction interval P_(r) is calculatedfor the next time duration x_(n+1) according to:

${{\Pr \left( {{{\overset{\_}{x}}_{n} - {T_{a}s_{n}\sqrt{1 + \left( {1/n} \right)}}} \leq x_{n + 1} \leq {{\overset{\_}{x}}_{n} + {T_{a}s_{n}\sqrt{1 + \left( {1/n} \right)}}}} \right)} = p},{where}$$\overset{\_}{x} = {{\overset{\_}{x}}_{n} = {\left( {x_{1} + x_{2} + \ldots + x_{n}} \right)/n}}$$s^{2} = {s_{n}^{2} = {\frac{1}{n - 1}{\sum\limits_{i = 1}^{n}\left( {x_{i} - {\overset{\_}{x}}_{n}} \right)^{2}}}}$

T_(a) is the

$100\left( \frac{1 + p}{2} \right){th}$

percentile of the t-distribution with n−1 degrees of freedom.Using the above example,

${\overset{\_}{x}}_{n} = {{\left( {7 + 8 + 6 + 7 + 14} \right)/5} = 8.4}$$s_{n} = {\sqrt{\frac{1}{5 - 1}\left\lbrack {\left( {7 - 8.4} \right)^{2} + \left( {8 - 8.4} \right)^{2} + \left( {6 - 8.4} \right)^{2} + \left( {7 - 8.4} \right)^{2} + \left( {14 - 8.4} \right)^{2}} \right\rbrack} = 3.209}$

For (5−1)=4 degrees of freedom and p=90%, i.e. a prediction interval of

${{100\left( \frac{1 + 0.9}{2} \right)} = {95\% \mspace{14mu} {percentile}}},{T_{a} = 2.132}$${\Pr \left( {{{\overset{\_}{x}}_{n} - {T_{a}s_{n}\sqrt{1 + \left( {1/n} \right)}}} \leq x_{n + 1} \leq {{\overset{\_}{x}}_{n} + {T_{a}s_{n}\sqrt{1 + \left( {1/n} \right)}}}} \right)} = p$${\Pr \left( {{8.4 - {(2.132)(3.209)\sqrt{1 + \left( {1/5} \right)}}} \leq x_{n + 1} \leq {8.4 + {(2.132)(3.209)\sqrt{1 + \left( {1/5} \right)}}}} \right)} = {90\%}$Pr (0.905 ≤ x_(n + 1) ≤ 15.90)

In this example, there is a 90% chance that the user will return to thelocation and capture another image approximately between one and 16 daysafter the last image was captured on the 29/9/2009. Depending on thetiming of the prediction interval relative to the current time and theduration window of the prediction interval, the user may consider thelocation as personal if there is a significantly high probability ofassociation between the user, the location and a window of time in thefuture.

The controller 622 can perform steps 210, 220, and 230.

FIG. 5A illustrates the scenario of a highly predictable image capturingpattern, e.g. weekly netball matches at a local sports groundparticipated by the child of a user. The prediction interval 510indicates a 90% chance that the user is associated with a particularlocation within a 2-day window. This suggests that there is a privacyconcern to the user if the prediction interval is still in the future,but this concern diminishes as more time passes beyond the predictioninterval. In this case, the user may be prompted to confirm whether thelocation of concern is in fact personal or not.

In contrast, FIG. 5B shows the scenario of an irregular image capturingpattern. The prediction interval 520 indicates a 90% chance that theuser is associated with a particular location within a 32-day window.Even if the prediction interval is still in the future, some users maybe comfortable to disregard the location as personal due to theinaccuracy of such a wide window. As privacy is to an extent aperception of the user, the prediction interval percentile and theduration window thresholds can be configured to match the comfort zoneof the user.

[Suppressing Metadata]

After identifying the images of concern, sensitive metadata values suchas geotag, timestamp, keywords, etc. needs to be suppressed beforesharing the images publicly. The embodiments of the step 160 forsuppressing selected sensitive metadata values are describedhereinafter. FIG. 3 shows in detail the step 160 of FIG. 1 forsuppressing selected metadata values by encryption. In step 310, theselected metadata values are encrypted. This may include the sensitivemetadata regarding privacy. In step 320, the encrypted metadata valuesare saved under different tags in the metadata. Preferably, the userwith access to the encryption key can retrieve the original metadatavalues transparently. In step 330, the selected metadata values underthe original metadata tag are replaced by different values to mitigateany privacy concern. Some examples of this range from assigning a blankvalue, introducing a deliberate error to the value, or reducing theresolution of the value, e.g. disclosing a location at a city level,disclosing a timestamp to the nearest year, etc. Processing then ends.

FIG. 4 shows in detail an alternative implementation of step 160 forsuppressing selected sensitive metadata values by storing the originalmetadata values in a separate data store. In step 410, selected metadatavalues are stored in an external sidecar data store that is not publiclyshared. Preferably, the storage organisation allows the user transparentaccess to the original metadata values using a unique reference of thecorresponding image. In step 420, the selected metadata values under theoriginal metadata tag are replaced by a different value, as in step 330.Processing then ends.

Also, the processes of FIGS. 3 and 4 may be combined so that selectedmetadata is encrypted, removed from the original metadata tag, replacedwith a different value, and the encrypted metadata value is storedexternally as in step 410 of FIG. 4.

The processing 105 carried out in steps 110, 120 and 130 is based on theuse of probability techniques. This particular embodiment has anadaptive characteristic such that a change in the user's image capturingpattern over a period of time may alter the result, e.g. a locationdetermined as personal previously may no longer be personal if no imageshave been captured at the location for long period of time. It will beapparent to those skilled in the art that other processing embodimentsbased on analysing the pattern of the image timestamps may be used toprovide other desirable characteristics.

One such alternative embodiment to perform the processing 105 isdescribed hereinafter with reference to FIG. 7. In step 710, arepresentative geolocation is associated with each group of imagescaptured at the same location as in step 110 of FIG. 1. In step 720, theimages associated with the same representative geolocation are groupedby date based on the timestamp of the images. This number typicallyindicates the number of visits or times present at the location. In step730, the corresponding location (representative geolocation) isclassified as personal if the number of visits is above a predeterminedthreshold. That is, if images have been captured at the same geolocationon multiple and different occasions, the geolocation is deemed personal.This embodiment provides a different characteristic as that of steps110, 120 and 130, such that the determination is static rather thanadaptive. That is, if a geolocation is determined as personal, theresult is not affected by future changes in the user's image capturingpattern.

Yet another alternative embodiment may involve the use of a machinelearning technique to perform supervised learning of timestamp patternsof images captured by different users at known personal locations. Aclassifier trained by the learning can then be used to classify whethera representative geolocation of another user's images is personal.

Another aspect of the invention is to address the suppression ofsensitive metadata of all images captured at strictly private locationssuch as private residences, schools, etc. more definitively. Althoughpossible, the classification of these definitively private locationsdoes not require the same personal location analysis, and theclassification does not require to be changed in the same manner aspersonal locations by adapting to the image-capturing pattern of theuser. Therefore, in addition to the process 100, the representativegeolocation of private places may be entered by the user explicitlyinstead of determined by the personal location analysis steps 110 to130. The user can specifying a geolocation as a personal location. Thisstep is different to the steps of the methods 105 and 100 of FIG. 1 anddoes not result from those steps. The means for entering privaterepresentative geolocations to augment the personal representativegeolocations include but not limited to the following: inputting textualaddresses, importing addresses from an existing contact list, manuallyentering or setting the current geolocation as a private representativegeolocation.

Another aspect of the invention is concerned with using the process 100without the immediate suppression of metadata 160. Instead, when adetermination is made that an image was captured at a personal location150, the image is flagged for further processing by another systemoutside the image capture system 600. This may be achieved by creatingand setting such flag in the metadata of the captured image or bypassing the flag as additional data along with the images to anothersystem.

Yet another aspect of the invention is concerned with using the process100 of FIG. 1, in particular steps 110 to 130, in an image managementsystem, such as a photo management application, to identify a subset ofimages captured at personal locations in order to allow the system tomitigate any privacy risks when sharing the images publicly.

INDUSTRIAL APPLICABILITY

The arrangements described are applicable to the computer and dataprocessing industries and particularly for the processing digitalimages.

Methods, apparatuses, and computer readable storage mediums forprocessing captured images having geolocations related to the capturedimages at the time of capture have been described. The foregoingdescribes only some embodiments of the present invention, andmodifications and/or changes can be made thereto without departing fromthe scope and spirit of the invention, the embodiments beingillustrative and not restrictive.

In the context of this specification, the word “comprising” means“including principally but not necessarily solely” or “having” or“including”, and not “consisting only of”. Variations of the word“comprising”, such as “comprise” and “comprises” have correspondinglyvaried meanings.

1. A method of processing captured images having geolocations related tothe captured images at the time of capture, said method comprising thesteps of: associating a representative geolocation with a group ofimages previously captured at or near the same location; and for therepresentative geolocation, determining whether the representativegeolocation is a selected place based on the pattern of the timestamp ofthe images.
 2. A method of processing captured images havinggeolocations related to the captured images at the time of capture, saidmethod comprising the steps of: associating a representative geolocationwith each group of images previously captured at the same location; foreach representative geolocation, determining, based on at least thetimestamp of the images associated with the representative geolocation,the probability that a further image will be captured at or near therepresentative geolocation; and for representative geolocations with adetermined probability above a predetermined threshold, associating therespective representative geolocation with at least one personal place.3. The method as claimed in claim 2, further comprising the step ofcapturing an image having a geolocation related to the captured image atthe time of capture.
 4. The method as claimed in claim 2 or 3, furthercomprising the step of suppressing at least one metadata value of acaptured image if the geolocation related to the captured image is at ornear at least one personal place.
 5. The method as claimed in claim 4,wherein said at least one metadata value is suppressed by at least oneof storing the metadata value in an external data store separately fromthe captured image and encrypting the metadata value.
 7. The method asclaimed in claim 2, further comprising the step of specifying ageolocation as a personal location.
 8. The method as claimed in claim 7,wherein specifying the geolocation of a personal location comprisesinputting textual addresses, or importing addresses from a contact list,or manually entering the geolocation.
 9. The method as claimed in claim3, wherein the captured image is captured using an image capturingapparatus adapted to acquire a geolocation related to the captured imageat the time of image capture.
 10. The method as claimed in claim 2,wherein the geolocations and at least the timestamps of the images arestored for further processing, said images being downloaded separately.11. An apparatus for processing captured images having geolocationsrelated to the captured images at the time of capture, said apparatuscomprising: a memory for storing data and a computer program; and aprocessor unit coupled to the memory for executing a computer program,said memory and said processor configured to process said capturedimages, the computer program comprising: computer program code means forassociating a representative geolocation with each group of imagespreviously captured at the same location; computer program code meansfor each representative geolocation, determining, based on at least thetimestamp of the images associated with the representative geolocation,the probability that a further image will be captured at or near therepresentative geolocation; and computer program code means forrepresentative geolocations with a determined probability above apredetermined threshold, associating the respective representativegeolocation with at least one personal place.
 12. The apparatus asclaimed in claim 11, further comprising computer program code means forsuppressing at least one metadata value of a captured image if thegeolocation related to the captured image is at or near at least onepersonal place.
 13. The apparatus as claimed in claim 12, wherein saidat least one metadata value is suppressed by at least one of storing themetadata value in an external data store separately from the capturedimage and encrypting the metadata value.
 14. The apparatus as claimed inclaim 11, further comprising an image capturing apparatus adapted toacquire a geolocation related to a captured image at the time of imagecapture.
 15. The apparatus as claimed in claim 11, wherein thegeolocations and at least the timestamps of the images are stored in thememory of the apparatus for further processing, allowing said images tobe downloaded from the apparatus and removed from the memory.
 16. Acomputer readable storage medium having recorded therein a computerprogram for processing captured images having geolocations related tothe captured images at the time of capture for execution by a processingunit, the computer program comprising: computer program code means forassociating a representative geolocation with each group of imagespreviously captured at the same location; computer program code meansfor each representative geolocation, determining, based on at least thetimestamp of the images associated with the representative geolocation,the probability that a further image will be captured at or near therepresentative geolocation; and computer program code means forrepresentative geolocations with a determined probability above apredetermined threshold, associating the respective representativegeolocation with at least one personal place.
 17. The computer readablestorage medium as claimed in claim 16, further comprising computerprogram code means for suppressing at least one metadata value of acaptured image if the geolocation related to the captured image is at ornear at least one personal place.
 18. The computer readable storagemedium as claimed in claim 16, wherein said at least one metadata valueis suppressed by at least one of storing the metadata value in anexternal data store separately from the captured image and encryptingthe metadata value.
 19. The computer readable storage medium as claimedin claim 16, wherein said images are captured using an image capturingapparatus adapted to acquire a geolocation related to a captured imageat the time of image capture.
 20. The computer readable storage mediumas claimed in claim 16, wherein the geolocations and at least thetimestamps of the images are stored for further processing, allowingsaid images to be downloaded.